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CO ! Abstract 
O ■ 

One way to incorporate systematic uncertainties into the calculation of confidence intervals is 
by integrating over probability density functions parametrizing the uncertainties. In this note we 

<D : 

Ph ■ present a development of this method which takes into account uncertainties in the prediction of 

background processes, uncertainties in the signal detection efficiency and background efficiency and 

■ allows for a correlation between the signal and background detection efficiencies. We implement 

> : 

CO ■ this method with the Likelihood Ratio (usually denoted as Feldman & Cousins) approach with and 

t-H ■ 

' without conditioning. 

cn : 

We present studies of coverage for the Likelihood Ratio and Neyman ordering schemes. In particu- 
lar, we present two different types of coverage tests for the case where systematic uncertainties are 

x : 

D , included. To illustrate the method we show the relative effect of including systematic uncertainties 

Qh' 

' the case of dark matter search as performed by modern neutrino telescopes. 

■ PACS numbers: 06.20.Dk, 95.55.Vj 

x : 
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I. INTRODUCTION 



A limit on , or a measurement of , a physical quantity at a given confidence level is usually 
set by comparing a number of detected events, n , with the number of expected events from 
the known background sources contributing to the physical process in question, n&. How 
'compatible' these numbers are determines how much room there is for new processes, ie., 
for a signal. How well do the number of observed events and expected background compare, 
strongly depends on the systematic uncertainties present in the measurement. Systematic 
uncertainties must, therefore, be taken into account in the limit or confidence belt calculation 
that is finally published. 

Traditionally, confidence limits are set using a Neyman construction This is a purely 
frequentist method. G. Feldman and R. Cousins [2| have proposed an improved method 
to construct confidence intervals based on likelihood ratios, a method allready known in 
statistics and originally described in |J. Still, this method is based on the original Ney- 
man construction, and needs to be extended to incorporate systematic uncertainties in the 
measurement. Along this line, a modification of the Neyman method that incorporates sys- 
tematic uncertainties in the experimental signal efficiency has been proposed by V. Highland 
& R. Cousins Q]. These authors use a "semi"- Bayesian approach where an average over the 
probability distribution of the experimental sensitivity (and its uncertainty) is performed. 
By construction, the method is of limited accuracy in the limit of high relative systematic 
uncertainties. 

Recently, an entirely frequentist approach has been proposed for uncertainty in the back- 
ground rat e predict ,,. T hat approach is based on a two-dimensional confidence belt 
construction and likelihood ratio hypothesis testing and treats the uncertainty in the back- 
ground as a statistical uncertainty rather than as a systematic one. 

The interest aroused recently in the High Energy Physics community about the many 
open issues on setting limits and quoting confidence levels is stresseded by the organization 
of devoted workshops on the subject. We refer the reader to the proceedings of the recent 

n q n 

workshops at CERN |6(, FERMILAB 7] and Durham [8( for a review of the status of the 
field. 

In this paper we extend the method of confidence belt construction proposed in ^| to 
include systematic uncertainties both in the signal and background efficiencies as well as 
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theoretical uncertainties in the background prediction. The proposed method allows as well 
to use newer ordering schemes. A recent attempt to include systematic uncertainty in the 
background prediction in a similar manner has been presented in . The paper is organized 
as follows. In section |TI] we give a short review of the confidence belt construction schemes 
which we will use. In section HTD we describe how to include the systematic uncertainties, in 
section IIVI we discuss how the confidence belt construction is performed and present some 
selected results. We compare the results of this method with other methods to include 
systematics in section |V| 

We introduce the tests of coverage performed in section ED and present an example based 
on data from the AMANDA neutrino experiment in section IV11I 

II. THE CONSTRUCTION OF CONFIDENCE INTERVALS 

The frequentist construction of confidence intervals is described in detail elsewhere jlfll |. 
Here we will give just a short review. 

Let us consider a Poissonian probability density function (PDF), p(n) s+b , for a fixed but 
unknown signal, s, in the presence of a known background with mean b. For every value of 
s we can find two values n\ and n 2 such that 



where I — a denotes the confidence level (usually quoted as a 100(l-a)% confidence interval). 
Since we assume a Poisson distribution, the equality will generally not be fulfilled exactly. 
A set of intervals [ni(s + b, a),ri2{s + b, a)] is called a confidence belt. Graphically, upon 
a measurement, n Q , the confidence interval [si,S2] is determined by the intersection of the 
vertical line drawn from the measured value n Q and the boundary of the confidence belt. 
This is illustrated in figure H The probability that the confidence interval will contain the 
true value s is 1 — a, since this is true for all s per construction. 

The choice of the ni and n 2 is, however, not unique to define the confidence belt. An 
additional criterion has to be applied. The choices originally proposed by Neyman are 
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FIG. 1: Illustration of the confidence belt construction. On the x axis are the possible 
experimental outcomes (number of events), on the y axis the parameter of the pdf (s). In 
this case a Poisson PDF was assumed. 



for central confidence intervals, and 



s+b 



1 — a 
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for upper confidence limits. This method presents certain drawbacks in the case of small 
samples and, in particular, can yield null results (in the sense that the algorithm gives no 
answer) in the case when no events have been observed. Also, the decision of quoting a 
measurement (that is, a central confidence interval) or an upper limit might not be straight- 



forward before performing an experiment. 



A. Likelihood ratio ordering 

To solve this problem, a modification of the Neyman method has been proposed 

aa 

that is based on a more rationalized ordering scheme of the elements in the sum in equation 
based on likelihood ratios. This approach automatically provides central confidence 
intervals when motivated and upper limits when necessary, therefore it is often denoted 
as the "unified approach" . Instead of using the choices given in the previous section, the 
following ordering scheme is applied in solving equation ([1)1: 

For each n the Sbest is found which maximizes the likelihood £(n) s+ b- In case of a simple 
Poissonian distribution with known background, Sbest is given by max(0,n — b). Then for a 
fixed s the ratio 

RM ^cZ^M (4) 

is computed for each n, and all n's are consequently ranked according to the value of this 
ratio. Values of n are included in the confidence belt starting with the n with the highest 
rank (largest Rc) and then decreasing rank until Y^n=n p(n) s+b = 1 — a. After the confidence 
belt has been constructed in this way, the confidence interval [si,S2] is found as described 
in the previous section. Note that this ordering principle is a standard method within the 
theory of likelihood ratio tests 

This approach has some undesired features as well. There is a background dependence 
of the upper limit in case of less events observed than expected from background. This can 
lead to situations where measurements with higher background give a better limit, a clearly 



111 ] proposed a solution to this problem which 



undesireable effect. B. Roe & M. Woodroofe 
we briefly describe next. 

B. Conditioning 

A variation of the classical method of constructing confidence belts is to use the fact that, 
given an observation n , it is known that the background can not have been larger than n 



itself. To incorporate this knowledge into the PDF, the authors in ll| have proposed the 
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following modification: 



n 



s+b 



if n < n Q 



n'=0 



p(n') b p( 



(5) 



n — n 



n'=0 



if n > n 



n'=0 



The likelihood ratio ordering can then be applied with this new PDF. Note that in this 
case the PDF is dependent on the number of observed events. This approach solves the 
background dependence of the upper limit: a limit set when no events are observed stays 
constant at a value of 2.44 independent of the expected background (which agrees with the 
result of the original likelihood ordering for no events observed and no expected background). 



However, this method does not satisfy all the requirements of proper coverage jl2j and has 
problems when applied to the case of a Gaussian distribution with boundaries 13|. An 
extension based on a Bayesian approach with tests of coverage can be found in 14|. 



III. THE INCLUSION OF SYSTEMATIC UNCERTAINTIES 

The way of incorporating systematic uncertainties into the confidence belt construction 
presented in this paper does not affect the particular ordering scheme. Instead, it takes into 
account the systematic uncertainties by assuming (or if possible determining) a PDF which 
parameterizes our knowledge about the uncertainties and integrating over this PDF. It has 
been noted that averaging over systematic uncertainties in itself is a Bayesian approach jj] 
. Therefore the method presented is refered to as "semi-Bayesian" , combining classical and 
Bayesian elements. We return to this point in section IVTl Usually, uncertainties are assumed 
to be described by a Gaussian distribution, which we will adopt for the remainder of this 
paper. The implementation, however, makes it easy to use other parametrizations for the 
uncertainties. 

We will refer in the following to the parameters with systematic uncertainties also as 
nuisance parameters. 

Two examples of how the PDF modifies if systematic uncertainties are present are the 
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following. In the case that the only uncertainty present is a theoretical uncertainty of the 
background process the PDF is modified to: 



Here b is the estimated background level, and g\> is the uncertainty in the background 
estimation. If, in addition to the theoretical uncertainty for background, there is the need 
to include the uncertainty in the signal detection efficiency the expression for q(n) s+ b might 
be extended to: 



where a e is the uncertainty in the detection efficiency expressed in relative terms with respect 
to the nominal efficiency. It is important to realize that the integration variables, here e' 
and b', are the possible "true" (but unknown) values of nuisance parameter. This indicates 
that this method is based on Bayesian statistics. 

IV. POLE: A GENERAL ALGORITHM FOR CONFIDENCE BELT CONSTRUC- 
TION 

The integrals © and (JZj) can be solved using different methods. We note, however, that 
they are examples of simplified cases. The most general experimental situation involves both 
an uncertainty in signal efficiency as well as in the background detection efficiency, which 
are usually correlated, and possibly an additional theoretical uncertainty in the background 
process prediction. We have developed an algorithm that takes these effects into account 
with the proper correlations between them. The algorithm performs a Monte Carlo inte- 
gration over the systematic uncertainties. It has been implemented as a FORTRAN program, 
POLE (POissonian Limit Estimator) Q]. In the examples used in this section a Gaussian 
distribution of uncertainties is assumed, but the algorithm makes it easy to implement PDFs 
other than Gaussian (see the next section for an example of using a different distribution). 
For the moment the code supports a Gaussian, flat and log-normal parametrization of the 
uncertainties. After determining the PDF through evaluation of the integrals, different or- 
dering schemes can be applied for the final calculation of the confidence belt. The results 
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presented here are mainly for the likelihood ratio ordering scheme with and without condi- 
tioning. We restrict ourselves to present systematic uncertainties of signal and background 
efficiencies separately to give a clear idea of the effect of varying a single variable at a time. 
Real applications usually combine those uncertainties. 

The confidence belt constructions have been performed using steps of 0.05 in signal ex- 
pectation and performing the construction up to a maximal signal expectation of 50 and 
a maximal number of detected events of 100. Including systematic uncertainties generally 
leads to a widening of the confidence belt. Figure El shows an example of a likelihood ratio 
confidence belt construction with and without uncertainty in the signal efficiency, where a 
background expectation 6 = 2 has been assumed. 

Examples for some resulting intervals are given in tables and Different combinations 
of number of observed events, no, and expected background, b, are given for different uncer- 
tainties in the signal and background efficiency. 

The width of the interval for two particular examples of observed events and expected back- 
ground as function of signal efficiency uncertainty and background uncertainty is shown in 
figure El Note that for low background expectation, the uncertainties in the background can 
be neglected (see also table |HJ). Figures 0] and El give more extended information on resulting 
intervals. 

An interesting case arises when there are significantly less events observed than expected 
from background and there is an uncertainty in the signal efficiency. In this case, the width 
of the confidence interval does not increase (see table ITTT]) . Note that if we use conditioning 
the effect disappears. The same can not be observed in the case where we only consider an 
increasing background uncertainty (see table 

A. Negative values of the nuisance parameters: Using a log - normal distribution. 

In experimental situations where the systematic uncertainties are high, a problem might 
arise due to the fact that sampling from a Gaussian PDF allows negative values. POLE is 
dealing with these cases truncating the Gaussian distribution and renormalizing the part 
above zero. 

We examine the effect of truncating the Gauss distribution by calculating the confidence 
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FIG. 2: 90% confidence belts obtained with POLE using the likelihood ratio ordering scheme 
and assuming different uncertainties in the signal efficiency. The inner band has been 
constructed assuming no uncertainty in the signal efficiency. The outer band represents the 
belt constructed with a signal efficiency uncertainty of 40%. The background expectation 
in this particular case was b = 2 

interval for different values of truncation point (see figure EJ). Considering a Gaussian distri- 
bution centered on one widh a = 40 %, a truncation at zero removes only 0.7 %. Figure El 
therefore indicates that effects on the confidence interval due to the truncation are negligible 
for all cases considered in this paper. 
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0: 3.90 


0: 4.00 






0.2 


0: 3.95 


0: 4.34 






0.3 


0: 4.10 


0: 4.75 






0.4 


0: 4.65 


0: 5.35 
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0: 5.40 


0: 5.30 






0.2 


0: 5.70 


0: 5.65 






0.3 


0: 5.95 


0: 6.20 






0.4 


0: 6.80 


0: 7.10 
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0: 6.60 


0: 6.60 






0.2 


0: 7.10 


0: 7.30 






0.3 


0: 7.75 


0: 7.85 






0.4 


0: 8.95 


0: 9.15 
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0.40: 7.95 


0.50: 8.05 






0.2 


0.40: 8.60 


0.50: 8.60 






0.3 


0.40: 9.55 


0.50: 9.65 






0.4 


0.40:11.15 


0.50:11.20 


6 


2 





1.10: 9.45 


1.10: 9.45 






0.2 


1.05,10.05 


1.05:10.10 






0.3 


1.05:11.50 


1.05:11.50 






0.4 


1.05:13.35 


1.05:13.35 



TABLE I: Examples of likelihood ratio 90% confidence intervals including systematic uncertainty 
in the signal efficiency and assuming no uncertainty in the background prediction. 

A PDF for the nuisance parameters extending to negative values or which falls off to zero 
discontinuously is certainly undesired from a conceptual point of view. We therefore test the 
behavior of the confidence interval if we replace the Gaussian distribution with a log-normal 
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0: 3.90 
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0.2 
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0: 3.95 


0: 4.25 






0.4 


0: 3.95 


0: 4.35 
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0: 5.40 


0: 5.30 






0.2 


0: 5.45 


0: 5.35 






0.3 


0: 5.45 


0: 5.45 






0.4 


0: 5.50 


0: 5.55 
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0: 6.60 


0: 6.60 






0.2 


0: 6.95 


0: 6.65 






0.3 


0: 6.95 


0: 6.80 






0.4 


0: 6.95 


0: 6.80 


5 


2 





0.40: 7.95 


0.50: 8.05 






0.2 


0.35: 7.95 


0.50: 8.10 






0.3 


0.30: 8.00 


0.50: 8.10 






0.4 


0.20: 8.20 


0.45: 8.15 


6 


2 


0. 


1.10: 9.45 


1.10: 9.45 






0.2 


1.05: 9.45 


1.10: 9.50 






0.3 


1.00: 9.50 


1.05: 9.50 






0.4 


0.95: 9.50 


1.00: 9.50 



TABLE II: Examples of likelihood ratio 90% confidence intervals including systematic uncertainty 
in the background expectation and assuming no uncertainty in the signal efficiency. 



distribution, which in the general form is given by: 



i ^ i n — fj, f 

q{x)w = e—^~ (8) 



We require the mean of the log-normal distribution to be the nominal value of the nuisance 
parameter and use the Gaussian standard deviation as before (the variance of the log normal 
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signal efficiency uncertainty background uncertainty. 



FIG. 3: Example of the dependence of the likelihood ratio confidence interval width on the sys- 
tematic uncertainties, with and without conditioning, as obtained with POLE. The left plot shows 
the width as a function of the uncertainty in signal efficiency assuming no additional uncertainty 
in background expectation. The right plot shows the width as a function of the background un- 
certainty. We have used n =17 and a background of 15 in constructing the plots. 

distribution will then be approximately the same). The confidence interval for Neyman and 
likelihood ratio ordering under these assumptions are shown for one particular example of 
number of observed events and expected background as a function of signal efficiency in 
figure The differences between using a Gaussian distribution and using a log-normal 
distribution are generally small, in our example less than ~ 2 %. The use of a log- normal 
distribution is implemented as an option in POLE. 



V. COMPARISON WITH THE X 2 METHOD. 

Since the ratio of the likelihoods is asymptotically x 2 distributed the approximation: 

A X Z = -2 x mm — (9) 

e L,[n , Sbest, tbest) 

is often used, see e.g. jlo| . 

Here for a given observation, n , the A\ 2 is calculated as a function of s and a cut on A% 2 is 
performed to obtain the confidence interval (e.g. A% 2 = 2.71 corresponds to 90 % confidence 
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0: 1.55 


0: 3.15 






0.2 


0: 1.55 


0: 3.35 






0.4 


0: 1.45 


0: 4.00 
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6 





0: 2.85 


0: 4.30 






0.2 


0: 3.20 


0: 4.60 






0.4 


0: 3.35 


0: 5.35 



TABLE III: Likelihood Ratio confidence intervals with systematic uncertainty in the signal effi- 
ciency and no uncertainty in the background expectation. Here two examples are shown where there 
are less events observed than expected background: The interval does not increase with increasing 
uncertainty if there are significantly less events observed than expected background. However, if 
expected background and number of observed events are comparable, the interval becomes larger. 
In case conditioning is applied, it grows larger in all cases. 

level for one degree of freedom). 

Figure |S1 illustrates the effect of including uncertainties on the resulting confidence inter- 
vals for the x 2 approximation as compared to the method proposed here. 
Generally, the x 2 approximation gives more conservative results than the pole method. Since 
- as we will see in the following sections - using the pole method leads to some over-coverage, 
this is clearly undesirable. 

VI. TESTS OF COVERAGE 

From a frequentist point of view, an algorithm is said to have the correct coverage if, given 
a confidence level 1 — a and a large number of repeated identical experiments, it provides 
correct answers in a fraction 1 — a of the cases, independent of the value of s. To test the 
coverage of the algorithm proposed in this paper, we perform the construction described 
in the previous sections for a large number of simulated experiments, where we predefine 
the true signal and background and then determine n by random sampling from a Poisson 
distribution. We then calculate how often the obtained confidence interval does not contain 
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6 





0: 1.55 


0: 3.15 






0.2 


0: 1.55 


0: 3.50 






0.4 


0: 2.64 


0: 3.85 
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6 





0: 2.85 


0: 4.30 






0.2 


0: 3.25 


0: 4.55 






0.4 


0: 4.60 


0: 5.55 



TABLE IV: Likelihood ratio confidence intervals with systematic uncertainty in the background 
expectation and no uncertainty in the signal efficiency. Here two examples are shown where there 
are less events observed than expected background: The confidence interval becomes larger with 
increasing uncertainty in the background expectation. 



the predefined s. We define the coverage ratio: 

^ false 



R 



ntot 



(10) 



Here, rif a i se denotes the number of simulated experiments in which the result of the algorithm 
does not contain the predefined s, and n tot denotes the number of simulated experiments 
performed. If we choose 1 — a to be 0.9, perfect coverage would mean R = 0.1, independent 
of signal expectation assumption. 

A value or R smaller than 0.1 means that the method over-covers. Expected coverage was 
studied mostl y in the context of Bayesian intervals, small number of events or including 
conditioning 3| 2^1 1^1 1 and without taking into account systematic uncertainties. Very 
recently, a study was presented considering coverage including systematic uncertainties |l2| • 
In the next sections, besides presenting coverage tests done for higher signal expectations 
without uncertainties, we will focus on the coverage of the methods if systematic uncertain- 
ties are included. 
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A. Coverage without systematic uncertainties 

We show an example of a plot of the coverage ratio (here using steps of 0.1 in signal space) 
in figure |§] for Neyman and likelihood ratio ordering. Both methods seem to over-cover for 
almost all cases (which is expected because of the discreteness of the Poisson distribution). 
There is no signal expectation dependence of the coverage ratio except for the "see-saw" 
behavior which again reflects the discreteness of the Poisson distribution. 

B. Coverage with systematic uncertainties 

Introducing systematic uncertainties in the calculation of confidence intervals and tests 
of coverage leads to the question of what is meant by a repeated experiment. If we adhere 
to the traditional definition in which an experiment is repeated with fixed parameters such 
as efficiency or background rate, the algorithm presented here will inevitably yield over- 
coverage. Figure El shows the mean coverage ratio (mean here taken over six different 
signal expectation assumptions) as a function of different systematic uncertainties. The 
over-coverage will not only be increasing with increasing uncertainties but also be dependent 
on the signal expectation (figure 1X1)1 . 

1. Bayesian Coverage. 

The over-coverage described in the previous section is a consequence of the fact that 
efficiencies and background are not random variables (there is a true but unknown fixed ef- 
ficiency and background rate) but they are treated as random variables in the construction 
of the confidence belt (equations (jOJ) and (|ZJ)). 

Thus, while in the construction we are using a PDF which is a convolution of a Poisson 
distribution with a Gaussian distribution, repeated measurements (with parameters fixed) 
will produce a Poisson distribution. 

However, one has to keep in mind, that the distribution obtained in this way is not the 
underlying one. To infer from the measured Poisson distribution the underlying one, the 
signal efficiency and the background have to be taken into account. In particular, if these 
parameters are uncertain, there will not be a single underlying Poisson distribution, but a 
set of distributions that are weighted with the probabilities of the possible different efficien- 
ts 



cies and backgrounds. In a way, we thus give different hypotheses different weights. To take 
this into account we modify the coverage test described in the previous section. Instead of 
drawing a measurement from Poisson distributions with predefined signal expectation and 
background, we draw the signal expectation and background prediction used in each sim- 
ulated experiment from Gaussian distributions centered on the predefined true signal and 
background, where the width of the Gaussian is the associated systematic uncertainty. The 
measurement is then produced by taking these new values as means for the final Poisson 
distributions. 

Since, by using this approach, we give different weights to different hypotheses, we call this 
modified coverage test Bayesian Coverage. In this way, the PDF used in the construc- 
tion and in the coverage test are consistent with each other, and the algorithm should, per 
construction, give the correct coverage (except for discreteness effects). In particular, the 
coverage defined in this way should be independent of the magnitude of the uncertainties 
present in the experiment. Figure El shows the mean Bayesian coverage for different un- 
certainties in the signal efficiency together with the frequentist coverage. The mean is here 
taken over the 29 points in signal expectation space which were tested. As expected the 
Bayesian coverage ratio is nearly constant. 

Thus, if we loosen the criteria on the definition of "repeated experiment" , allowing the "true" 
(unknown) efficiencies vary for each experiment repitition , the method has the desired sta- 
tistical property of correct coverage. 

2. Remark on the choice of ensemble. 

In the previous subsection we consider an ensemble in which the true value of the nui- 
sance parameter, corresponding to e' in equation is varied in each of the members of the 
ensemble. We find, as expected, that the POLE- method fulfills the requirement of correct 
coverage with respect to this ensemble. 

However, it can be argued, that this ensemble does not describe the situation usually encoun- 
tered in experimental physics. The systematic uncertainty is a measurement uncertainty, i.e. 
not the true value of the nuisance parameter changes in each experiment but the measured 
one. 

Studies for a few cases indicate, that with respect to such an ensemble the POLE method 



16 



leads to moderate over-coverage [17[ 



r3, H- 



VII. LIMITS ON HIGH ENERGY COSMIC NEUTRINOS. 

In experimental situations where the systematic uncertainties are negligible small, limits 
or central confidence intervals can be calculated without evaluating the effects of the former. 
In more general situations, including systematic uncertainties in the calculation of the limits 
is essential, since it is a way of incorporating the real sensitivity of a given experiment to 
the quantity being measured. 

In this section we will consider two real examples taken from published results of the 
AMANDA neutrino telescope, where the program POLE was used to include the systematics 
in the final results. The AMANDA collaboration has published recently 90% confidence 
limits on the difuse flux of cosmic electron neutrinos and of neutrinos all flavours in the 
energy range between 5 TeV and 300 TeV The analysis revealed zero events with an 
expected background from atmospheric neutrinos of 0.01 events. The systematic uncertainty 
in the signal efficiency for this analysis was ~25%, determined from Monte Carlo studies. 
On top of that, the current theoretical systematic uncertainty in the atmospheric neutrino 



flux prediction in the energy range relevant to the analysis is about 30% [23j, which has to 
be taken into account as well. The effect of including the systematic uncertainties is shown 
in figure 12. The effect is to worsen the limits by about 10%. 

Another example worth noticing are the results published by the same collaboration on 
or supersymmetric dark matter in the form of weakly interacting massive particles, 



searches 

WIMPs 25]. In this analysis the uncertainties in signal efficiency range from 10% to 25% 
depending on the assumed signal spectrum and, additionally, an uncertainty in the back- 
ground detection efficiency, estimated to be 20%, has to be taken into account in this case. 
A further complication to include these uncertainties in the calculation of the final limits 
arises since the efficiencies in signal and background detection are correlated. Moreover, the 
mentioned theoretical uncertainty on the overall normalization of the atmospheric neutrino 
flux has to be added. In figure 13 we show the limits to the muon flux from the center 
of the earth as a function of WIMP mass. The full lines show the limits for two diferent 
assumptions on the signal spectra, and the dashed lines the corresponding limits without 
including systematic uncertainties in the calculations. 
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For the purpose of illustration, we show in figure 14 the effect of including systematic 
uncertainties in the limit calculation for five different values (dashed lines). From bottom 
to top: a 20% uncertainty in background expectation only, and a 10%, 20% 30% and 40% 
uncertainty in signal efficiency (on top of the mentioned 20% background uncertainty). The 
absolute scale of the plot is arbitrary since we are just interested in showing the relative 
effect of the inclusion of systematic uncertainties in the limit calculation, with respect to 
the no-systematics case (full line). The figure shows the importance of correctly evaluating 
systematics and including them in the final result, since the effects can be important. 

VIII. CONCLUSIONS 

In this note we present a Monte Carlo algorithm for introducing systematic uncertainties 
in the evaluation of classical confidence intervals which allows to include uncertainties 
in the background prediction, in the background detection efficiency and in the signal 
detection efficiency, and correlations between them, by integrating over the (assumed) 
PDFs of these parameters. We apply the method for a Poisson process with background 
under the assumption of a Gaussian PDF describing the uncertainties. We present results 
where the construction has been performed using likelihood ratio ordering with and without 
conditioning. 

Generally, the introduction of systematic uncertainties leads to an increase in confidence 
interval width. However, an interesting result is that likelihood ratio (as well as Neyman) 
confidence intervals which take into account the systematic uncertainty in the signal 
efficiency do not become larger with larger uncertainty, in the case that significantly 
less events have been observed than expected background. With respect to an ensemble 
with strictly identical experiments, introducing systematic uncertainties in the presented 
manner inevitably leads to over- coverage, increasing with the magnitude of systematic 
uncertainties. However, we show that with respect to an ensemble where the systematic 
uncertainties are taken into account in the coverage test by varying their true assumed 
values in each pseudoexperiment, the method presented here provides over-coverage only on 
the level already present due to the discreteness of the Poisson distribution. Both ensembles 
are ideal ensembles and have to be seen as approximations to the ensemble encountered 
in experimental physics. In summary, the algorithm presented here provides a practical 
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and flexible way to quantitatively take into account systematic uncertainties present in 
experimental situations in the calculation of confidence intervals. 
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FIG. 4: Likelihood ratio upper limit as a function of signal efficiency uncertainty for expected 
backgrounds = 0, 2, 4, 6. 8 and 10. Cases with number of observed events significantly less than 
expected background have been omitted. 
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FIG. 5: Likelihood ratio upper limit as a function of background efficiency uncertainty for expected 
backgrounds = 0, 2, 4, 6. 8 and 10. Cases with number of observed events significantly less than 
expected background have been omitted. 
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FIG. 6: The relative change of the likelihood ratio interval width as a function of fraction 
of Gaussian removed. In this example, n Q = 4 and 6 = 4 have been assumed. 
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FIG. 7: Likelihood ratio confidence interval width as a function of signal efficiency un- 
certainty for a Gaussian and a log- normal distribution with mean at 1. In this example, 
n Q = 4 and 6 = 4 have been assumed and the Gaussian was truncated at zero. 
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FIG. 8: The likelihood ratio confidence interval as calculated by pole and using the x 2 
approximation. Number of observed events is 10 (left panel) and 1 (right panel). The 
background was assumed to be zero in both cases. 
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FIG. 9: Coverage ratio as a function of different signal expectation assumptions. Left plot: Likeli- 
hood ratio ordering. Right plot: Neyman ordering. The thick line gives the line of perfect coverage, 
the thinner lines denote the measurement precision of this ratio we can achieve with 10000 simu- 
lated experiments (taken as 1 a of a binomial distribution). A constant background expectation 
of b= 10 has been assumed. 
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FIG. 10: Mean coverage ratio as a function of background uncertainty (left plot) and signal effi- 
ciency uncertainty (right plot). Here the mean is taken over six signal expectation assumptions 
between 12 and 42. The background expectation was taken to be constant b = 12. All other 
uncertainties than the one displayed were assumed to be zero. 



26 




FIG. 11: Signal expectation dependence of FIG. 12: Bayesian mean coverage ratio as a 

the coverage ratio. Here for the case where function of the uncertainty in signal efficiency, 

signal efficiency uncertainty is 30 %. The mean is here taken over 29 signal expec- 

tation values. For comparison the frequentist 
result using the same signal expectation as- 
sumptions has been included. 



27 







>" 






O 






£ 






X) 

< 

"O 






LJ 
~D 






\ 

-& -5 

LU 10 


\ \ AMANDA fi; 4- TA 






\ AMANDA [v, + i>,) 






E, (GeV) 


FIG 


13: Limit on the flux from cosmic 


neu- 



trinos of all flavors and electron neutrinos as 



presented in 




A signal efficiency uncer- 



tainty of 25 % and 30 % in backround predic- 
tion lead to an increase of the upper limit by 
about 10 %. 
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FIG. 14: Comparison of the effect of sys- 
tematica on the limit on the neutrino-induced 
muon flux from the center of the Earth from 
neutralino annihilation. The solid line repre- 
sents the current limit set by the AMANDA 
collaboration, the dashed line is the limit 
without including the systematic uncertain- 
ties present in their analysis (figure taken 



from 



251 ] . where more details of the analysis 



can be found.) 
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FIG. 15: Effect of including uncertainties on a 
generic WIMP limit. The solid line represents 
the limit calculation without any experimen- 
tal systematic uncertainties. The dashed lines 
represent (from bottom to top): a 20% uncer- 
tainty in the background expectation only, a 
10%, 20%, 30% and 40% uncertainty on the 
signal efficiency on top of the 20% background 
uncertainty. Additionally, a 30% uncertainty 
in the theoretical atmospheric neutrino back- 
ground expectation has been assumed in all 
cases. The absolute scale is arbitrary 
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